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5 TECHNICAL FIELD OF THE D^JVENTION 

This invention relates in general to redundant arrays of disks, such as RAID 
(Redundant Array of Independent Disks) sets. More specifically, the invention relates 
to an apparatus and method for providing virtual storage volumes, particularly very 
large virtual storage volumes (e.g,, 100 Gigabytes (GB) or more), using redundant 
1 0 arrays of disks, such as RAID sets. 

BACKGROUND OF THE INVENTION 
Some computer software appUcations are organized according to what is 
referred to as a "single volume architecture," meaning that they store data in a single 

15 data file that resides on a single volume. This "volume" may be a physical volume, 
such as a disk drive, or it may be a virtual volume, such as a RAID set. The 
Exchange® e-mail program provided by Microsoft Corporation of Redmond, 
Washington is an example of such a single-volume-architecture application. 

In some cases, the single volume architecture of a particular software 

20 application can be problematic, because the size of the data file the application needs 
to store on a single volume exceeds the capacity of the volxxme. For example, 
implementations of Microsoft's Exchange® e-mail program in large organizations 
having many e-mail users can require a single-volume storage capacity exceeding 
100GB, which is greater than many conventional volumes, physical or virtual, can 

25 provide. Although it is possible to solve this problem by changing a single-volume- 
architecture application into a multi-volume-architecture application so that it saves 
data in multiple files spread across multiple volumes, such efforts can be prohibitively 
time-consuming and expensive. 

Accordingly, there is a need in the art for a very large virtual storage volume 

30 having the storage capacity necessary to meet the needs of a single- volume- 
architecture software application, such as Microsoft's Exchange® e-mail program. 
Preferably, such a storage volume should have built-in disaster tolerance capabilities 
through the use of remote mirroring or other techniques in order to ensure the integrity 
of its stored data. In addition, such a storage volume should preferably have cloning 
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capabilities so that data backup can occur off-line without interrupting read/write 
access to the data. 



SUMMARY OF THE INVENTION 
5 An inventive apparatus for providing a very large storage volume includes a 

plurality of disks and a local back-end controller that organizes and presents the disks 
as redundant arrays of disks (e.g., RAID-5 sets). Also, a local front-end controller 
stripes the redundant arrays of disks and presents the striped arrays as a very large 
storage volume. 

10 To provide local redundancy, another plurality of disks and an associated 

back-end controller can be provided, in which case the local front-end controller 
forms mirror sets from the redundant arrays of disks presented by both back-end 
controllers. In addition, a further plurality of disks and an associated back-end 
controller can be provided to enable off-line backup of the data stored on the volume 

15 by cloning the data onto the disks, and then using the disks as the data source for off- 
line backup. Also, a still ftirther plurality of disks and an associated back-end 
controller can be provided at a remote location to protect against disasters occurring at 
the primary location (commonly referred to as "disaster tolerance"). The disks and 
back-end controllers providing cloning capabilities and disaster tolerance can be 

20 incorporated into the mirror sets formed by the local front-end controller. Further, 
spare disks can be provided on any or all of the back-end controllers to allow 
restoration of redundancy after the loss of any particular disk. 

If, for example, the disks each have 9.1GB of storage capacity and the local 
back-end controller organizes the disks into eleven, six-member RAID-5 sets, then the 

25 very large storage volume has a storage capacity in excess of 500GB, which should be 
adequate for most single-volume architecture programs. In addition, the redundancy 
restoration capabilities provided by the spare disks, the parity associated with RAID-5 
sets, and the mirroring ensures the integrity of the data stored on the very large storage 
volume. 

30 In another embodiment of this invention, the apparatus described above can be 

incorporated into an electronic system that also includes a host computer. 

In a further embodiment of this invention, data is stored on a plurality of disks 
by organizing the disks into a plurality of redundant arrays of disks. The redundant 



arrays of disks are striped together to form a virtual volume, and the data is then 
written to the virtual volume. 

In still another embodiment of this invention, data is again stored on a 
plurality of disks by organizing the disks into a plxorality of redxmdant arrays of disks. 
5 Mirror sets are formed from the redundant arrays of disks, and these mirror sets are 
then striped together to form a virtual volume. The data is then written to the virtual 
volume. 

BRIEF DESCRIPTION OF THE FIGURES 
10 Figures lA and IB is a diagram illustrating the organization of a very large 

volxme constructed in accordance with this invention; and 

Figure 2 is a block diagram illustrating the very large volume of Figures lA 
and IB. 

15 DETAILED DESCRIPTION OF THE ILLUSTRATED EMBODIMENTS 

As shown in Figures 1 A and IB, a 500.5GB very large volume 10 constructed 
in accordance with this invention is organized so as to comprise a RAID-0 stripe set 
having eleven, 45.5GB RAID-1 mirror sets Ml-Mll as members. Of course, it will 
be understood by those having skill in the technical field of this invention that 

20 although the invention will be described with respect to a very large volume having a 
500.5GB storage capacity, the invention is not limited to any particular storage 
capacity. In addition, it will be understood that the invention is not limited to the use 
of any particular redundant array technology (e.g., RAID) and, consequently, is not 
limited to the use of any particular RAID levels (e.g., RAJD-0, RAID-1). Also, it will 

25 be understood that the invention may include more or less than the eleven mirror sets 
Ml-Mll, and that the individual mirror sets Ml-Mll may be larger or smaller in size 
than the 45.5GB described here. 

As used herein, a "RAID-0 stripe set" will be understood to refer to a virtual 
volume comprised of two or more member disks or volumes across which "stripes" of 

30 data are stored. Also, as used herein, a "RAID-1 mirror set" will be understood to 
refer to a virtual volume comprised of two or more member disks or volumes, each of 
which contains an identical copy of the data stored in the mirror set. 
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The mirror set Ml, for example, comprises five, 45.5GB RAID-5 sets PLl, 
RLl, CI, PRl, and RRl as members. Similarly, the mirror set Mil comprises five, 
45.5GB RAID-5 sets PLll, RLll, Cll, PRll, and RRll as members. For purposes 
of clarity, the RAID-5 set members of the mirror sets M2-M10 are illustrated but not 
5 labeled. 

Of course, it will be understood that the members of the mirror sets Ml-Mll 
can be other than RAID-5 sets (e.g., RAID-3 or RAID-4 sets). Also, as used herein, a 
"RAID-5 set" will be understood to refer to a virtual volume comprised of three or 
more independently accessible member disks or volumes having redundancy 

1 0 protection through parity information distributed across its members. 

The RAID-5 sets PLl-PLll comprise the primary local storage copy of the 
data stored in the very large volxune 10, which means that they are the primary 
location to which the data is written and from which the data is read. Also, the RAID- 
5 sets RLl-RLll comprise a redundant local storage copy of the data, which provides 

15 mirroring-type redundancy for the stored data. In addition, the RAID-5 sets Cl-Cll 
comprise a cloning storage copy of the data, which is convenient for use in 
performing off-line data backups without interrupting read/write activities to the very 
large volume 10. Disaster tolerance is provided by the RAID-5 sets PRl-PRll, 
which comprise a primary remote storage copy, and the RAID-5 sets RRl-RRll, 

20 which comprise a redundant remote storage copy. Of course, it should be imderstood 
that embodiments of this invention may exclude the redundancy provided by the 
RAID-5 sets RLl-RLll, the cloning capability provided by the RAID-5 sets Cl-Cll, 
or the disaster tolerance provided by the RAID-5 sets PRl-PRll and RRl-RRll. 

The RAID-5 set PLl, for example, comprises six, 9.1GB physical disks 12 

25 distributed across six SCSI busses busl-bus6 of a back-end controller {see Figure 2). 
Similarly, the RAID-5 set PLll comprises six, 9.1GB physical disks 14 distributed 
across the six SCSI busses busl-bus6. In addition, six, 9.1GB spare physical disks 16 
seamlessly replace any failing disks on any of the busses busl-bus6 by rebuilding the 
data stored on failing disks from parity data, thereby restoring redundancy after a disk 

30 failure. 

As described herein, the very large volume 10 has a high degree of 
redundancy. If the bus3 physical disk 12 fails, for example, it is replaced by the bus3 
spare disk 16 by using parity data to rebuild the data stored on the failing bus3 



physical disk 12 onto the replacement bus3 spare disk 16. If bus3 itself fails, for 
example, the parity redundancy in the RAID-5 sets PLl-PLll regenerates the data 
stored on the failing bus3. If the back-end controller (see Figure 2) associated with 
the RAID-5 sets PLl-PLll fails, for example, the redundant local storage copy, 
5 comprised of the RAID-5 sets RLl-RLll, provides redundancy. Finally, if the front- 
end controller (see Figure 2) associated with the primary and redundant local storage 
copies and the cloning storage copy fails or is destroyed (e.g., due to a disaster), the 
primary remote storage copy, comprised of the RAID-5 sets PRl-PRll, and the 
redundant remote storage copy, comprised of the RAID-5 sets RRl-RRll, provide 
1 0 redundancy. 

As shown in a block diagram in Figure 2, the very large volume 10 is 
connected to a local host computer 20 that reads data from, and writes data to, the 
volume 10 via a local front-end controller 22 that acts as a mirroring and striping 
engine. In other words, the controller 22 forms the mirror sets Ml-Mll (see Figxires 

15 lA and IB) and then stripes them so as to present them to the local host computer 20 
as the very large volume 10. 

The primary local storage copy comprises physical disks 24 (which include 
disks 12, 14, and 16 of Figures lA and IB) connected to a back-end controller 26. 
The controller 26 acts as a RAID-5 engine by forming the disks 24 into the RAID-5 

20 sets PLl-PLll (see Figures lA and IB) and presenting the sets PLl-PLll to the 
front-end controller 22 as members. Similarly, the redundant local storage, clone, 
primary remote storage, and redundant remote storage copies comprise physical disks 
28, 30, 32, and 34, respectively, connected to back-end controllers 36, 38, 40, and 42, 
respectively, that act as RAID-5 engines by forming the disks 28, 30, 32, and 34 into 

25 the RAID-5 sets RLl-RLll, Cl-Cll, PRl-PRll, and RRl-RRll and presenting 
these sets to front-end controller 22 as members. 

In addition, the very large volume 10 is connected to a remote host 
computer 44 that reads data from, and writes data to, the volume 10 via a remote 
front-end controller 46 that acts as a mirroring and striping engine for the primary and 

30 redvindant remote storage copies. The local and remote host computers 20 and 44 are 
connected via a network interconnect 48, such as the internet or a dedicated network 
line. 
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In an alternative embodiment, the front-end controller 22 can be configxired to 
present a 273GB vmit and a 227.5GB unit to the local host computer 20, rather than 
the single 500.5GB unit described above. Of course, it shoiild be understood that the 
front-end controller 22 can, in fact, be configured in a multitude of ways to group the 
5 mirror sets Ml-11 into between one and eleven total units, each potentially ranging in 
size from 45.5GB to 500.5GB. Further, it should be understood that the front-end 
controller 22 can be configured (typically using software) to partition the striped 
mirror sets Ml-11 into an infinite number and size of units. 

In addition, it should be understood that although this invention has been 
10 described with reference to an embodiment having two levels of stacked controllers, 
the invention is not Umited to the two levels described. Rather, the invention includes 
within its scope any number of levels of stacked controllers. 

Although this invention has been described with reference to particular 
embodiments, the invention is not limited to these described embodiments. Rather, 
15 the invention is Umited only by the appended claims, which include within their scope 
all equivalent devices and methods that operate according to the principles of the 
invention as described. 
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CLAIMS 

1 1 . An apparatus for providing a virtual volume, the apparatus comprising: 

2 a plurality of disks; 

3 a back-end controller coupled to the disks for organizing and presenting the 

4 disks as a plurality of redundant arrays of disks; and 

5 a jfront-end controller coupled to the back-end controller for striping the 

6 redundant arrays of disks and presenting the striped arrays as a virtual volume. 

1 2, The apparatus of claim 1 v^herein the plurality of disks includes one or 

2 more spare disks. 

1 3. The apparatus of claim 1 wherein the back-end controller includes a 

2 plurality of busses, each coupled to one and only one of the disks associated v^th each 

3 of the redundant arrays of disks. 

1 4. The apparatus of claim 1 wherein the back-end controller comprises a 

2 RAID engine for presenting the disks as a plxirality of RAID sets. 

1 5. The apparatus of claim 4 wherein the RAID engine comprises a RAID 

2 engine for presenting the disks as a plurality of RAID-5 sets. 

1 6. An apparatus for providing a virtual volume, the apparatus comprising: 

2 a plurality of disks; 

3 a RAID engine coupled to the disks for organizing and presenting the disks as 

4 a plurality of RAID sets; and 

5 a striping engine coupled to the RAID engine for receiving the RAID sets as 

6 members, striping the member RAID sets, and presenting the striped RAID sets as a 

7 virtual volume. 

1 7. The apparatus of claim 6 wherein the RAID engine comprises a 

2 RAID-5 engine. 
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1 8. An apparatus for providing a virtual volume, the apparatus comprising: 

2 a plurality of back-end controllers, each configured to organize and present X 

3 N-member RAID sets, and each having N busses capable of supporting X+1 disks 

4 each; 

5 a plurality of groups of X+1 disks, each group being coupled to one of the 

6 back-end controller busses; and 

7 a local front-end controller coupled to the back-end controllers for receiving 

8 the RAID sets as members, striping the member RAID sets, and presenting the striped 

9 RAID sets as a virtual volume. 

1 9. The apparatus of claim 8 wherein the local front-end controller is 

2 configured to generate mirror sets from the RAID sets received as members from 

3 different back-end controllers, to stripe the mirror sets, and to present the striped 

4 mirror sets as the virtual volume. 

1 10. The apparatus of claim 8 wherein the plurality of back-end controllers 

2 includes primary local, redundant local, cloning, primary remote, and redimdant 

3 remote back-end controllers. 

1 11. The apparatus of claim 8 further comprising a remote front-end 

2 controller coupled to at least some of the back-end controllers for receiving RAID sets 

3 as members, striping the member RAID sets, and presenting the striped RAID sets as 

4 the virtual volxmie. 

1 12. The apparatus of claim 11 wherein the remote front-end controller is 

2 configured to generate mirror sets from the received RAID sets, to stripe the mirror 

3 sets, and to present the striped mirror sets as the virtual volume. 
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1 1 3 . An electronic system comprising : 

2 a computer; and 

3 an apparatus coupled to the computer for presenting a virtual volume to the 

4 computer, the apparatus including: 

5 a plurality of disks; 

6 a back-end controller coupled to the disks for organizing and 

7 presenting the disks as a plurality of redundant arrays of disks; and 

8 a front-end controller coupled to the back-end controller for striping 

9 the redundant arrays of disks and presenting the striped arrays as the virtual volume. 

1 14. A method of storing data on a plurality of disks, the method 

2 comprising: 

3 organizing the disks into a plurality of redundant arrays of disks; 

4 striping the redundant arrays of disks together to form a virtual volume; and 

5 writing the data to the virtual volume. 

1 15. The method of claim 14 wherein the act of organizing the disks 

2 comprises organizing the disks into a plurality of RAID sets. 

1 16. The method of claim 15 wherein the act of organizing the disks 

2 comprises organizing the disks into a plurality of RAID-5 sets. 

1 17. The method of claim 14 wherein the act of organizing the disks 

2 includes: 

3 providing one or more back-end controllers, each having a plurality of busses; 

4 and 

5 coupling the disks to the back-end controller busses so that each bus is coupled 

6 to no more than one disk from each redundant array of disks and each bus is coupled 

7 to a spare disk. 
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1 18. A method of storing data on a plurality of disks, the method 

2 comprising: 

3 organizing the disks into a plurality of redundant arrays of disks; 

4 forming mirror sets from the redundant arrays of disks; 

5 striping the mirror sets together to form a virtual volume; and 

6 writing the data to the virtual volume. 

1 19. The method of claim 18 wherein the act of organizing the disks 

2 comprises organizing the disks into a plurality of RAID sets. 

1 20. The method of claim 19 wherein the act of organizing the disks 

2 comprises organizing the disks into a plurality of RAID-5 sets. 
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ABSTRACT OF THE DISCLOSURE 
A very large virtual volume (e.g., in excess of 500GB) is formed by 
distributing the disks in eleven, six-disk RAID-5 sets across the six busses of a 
primary local back-end controller. A spare disk is provided on each of the six busses. 
Each RAID-5 set is protected from the failure of a single disk by the spare disks on 
the busses, which can use the parity data stored in a RAID-5 set to rebuild the data 
stored on a failing disk and thereby restore redundancy to the RAID-5 set. Each 
RAID-5 set is also protected from the failure of a bus by the parity inherent in 
RAID-5. The RAID-5 sets are striped by a front-end controller connected to the 
primary local back-end controller, and the striped RAID-5 sets are presented to a host 
computer as a very large virtual volume. If the individual disks are 9.1GB in size, the 
size of the very large virtual volume can reach 500.5GB. If desired, additional groups 
of eleven, six-disk RAID-5 sets can be formed on additional back-end controllers for 
purposes of redundancy, cloning (which generates a copy of the data that can be used 
for off-line backup without interrupting read/write activities on the virtual volume), 
and disaster tolerance through remote storage. These additional groups of RAID-5 
sets, along with the RAID-5 sets from the primary local back-end controller, can be 
formed into mirror sets by the front-end controller, which then stripes the mirror sets 
and presents the striped mirror sets to the host computer as the very large virtual 
volxmie. 
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Same ^ /x f\ , 


FULL NAME OF FIFTH JOINT INVENTOR 
Rvan J. Johnson 


INjl^mTOR'S SIGNATURE 


DATE , ^ 


RESIDENCE (7 J // 
14270 Timberedse lane, Colorado Springs, Colorado 80921 (/ 


CmZEUffSHI?^ 
USA 


POST OFFICE ADDRESS 

Same ^ . 


FULL NAME OF SIXTH JOINT INVENTOR 
Bert Martens 


INVENTOR'S SIGNATURE 


DATE 

-yyn 


RESIDENCE 

685 Calumet. Monument, Colorado 80132 


CITIZENSHIP 

USA 


POST OFFICE ADDRESS 

Same . . 


FULL NAME OF SEVENTH JOINT INVENTOR 
Karen E. Workman 


INVENTOR'S SIGNATURE 


DATE 

6-3-9^ 


RESIDENCE 7 / 
12655 Forest Green Dr., Elbert, Colorado 80106 ^ 


CITIZENSHIP 
USA 


POST OFFICE ADDRESS 

Same ^.^ . 


FULL NAME OF EIGHTH JOINT INVENTOR 
Sffi^n G. Elkington 


INVENTOR'S SIGNATURE 


DATE 


R^IDENCE ^ 
3W5 Black Canvon Road, Colorado Springs, Colorado 80904 


CITIZENSHIP 

USA 


nkBT OFFICE ADDRESS 

S^e . 


ipLL NAME OF NINTH JOINT INVENTOR 
I^fthard F. Lary 


INVENTp^'S SK3TVATWRE yn 


DATE . 


RESIDENCE /y 
IjgSO Summit Point Ct., Colorado Springs, Colorado 80919 ^ 


CITIZENSHIP 
USA 


POST OFFICE ADDRESS 

Same 


FW.L NAME OF TENTH JOINT INVENTOR 
Jjssse Yandell 


INVENTOR'S SIGNATmE 


DATE 


Rl^IDENCE ^ / 
Mio Fredricksburg Dr., Colorado Springs, Colorado 80922 


CITIZENSHIP 
USA 


P^T OFFICE ADDRESS ^ j 
Same . / 1 ^ /> / / 


FULL NAME OF ELEVENTH JOINT INVENTOR 
Stephen Sicola 


INVENTOR'S SIGTjfATURE / // 


DATE 1 


RESIDENCE *^ XT (j //^ ^ 
20430 Indi Dr., Monument, Colorado 80132 // 


CITIZENSHIP 

USA 


POST OFFICE ADDRESS 
Same 


FULL NAME OF TWELTH JOINT INVENTOR 
Roger Oakey 


I^^^DR'S^NATURE 


DATE , 

3(31 im 


RESIDENCE 

14105 Pine Glen Drive W., Colorado Springs, Colorado 80908 


CITIZENSHIP 


POST OFFICE ADDRESS 
Same 
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nAR-09-99 TO 01:51 PH 1668 



FAX NO. 1 



P. 02/05 



Express Mail No,; EL280219036US 



IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 



Appticant/Pafentse: 
Bruning, et al. 
Serial No.: 



Date Filed: Herewith 



§ 
§ 
§ 
§ 
§ 
§ 
§ 
§ 
§ 
§ 



Attorney File No.: 34309,830054.000 
Digital Docket No.: PD26112 



For: APPARATUS AND METHOD FOR PROVIDING 
VERY LARGE VIRTUAL STORAGE VOLUi\^ES USING 
REDUNDANT ARRAYS OF DISKS 

POWER OF ATTORNEY BY ASSIGNEE 



Under the provisions of 37 C,F,R. § 371, the undersigned assignee of record of the entire interest in the 
above-identified patent/patent application by virtue of an assignment recorded (check as applicable); 

S Concurrently Herewith 

□ Date Recorded 

□ Reel ^Frame ^ , 

elects to conduct the prosecution of the application/maintenance of the patent to the exclusion of the inventof(s). 
The undersigned hereby declares that she has reviewed the above-referenced assignment and hereby declares 
that, to the best of her knowledge, title is In the Assignee, and further declares that all statements made herein of 
her own knowledge are true and that all statements made on Information and belief are believed to be true. The 
assignee hereby revokes any previous powers of attorney and appoints the following to prosecute this 
application/maintain this patent and transact all business in the Patent and Trademark Office connected 
therewith: 



(Prosecuting Attorney List) 

Irene Kosturaki$, Reg. No. 33J24 
Rich Lange, Reg. No. 27,296 
Barry Blount, Reg. No/35,069 
Sarali T. Harris, Reg. No. 35,891 
Josepin Arrambide, Reg. No. 39,589 
Keith Lutsch, Reg. No, 31,851 
Francis A. Sirr, Reg, No. 17,265 
Earl C. Hancock, Reg, No. 19,472 
William J. Kubida, Reg, No, 29,664 
John R. Wahl, Reg. No, 33,044 



Robert H. Kelly, Reg. No. 33.922 
Stuart T. Langley, Reg. No, 33,940 
Robert 0, Crouch, Reg, No. 34,805 
Carol W. Burton, Reg, No. 35,465 
Elizabeth S. Cohen, Reg. No. 36,670 
Lee R. Osman, Reg, No. 38,260 
James A. Pinto, Reg, No. 40,774 
Earl C. Hancock, Reg. No. 19,472 
F.A, "Sandy" Sirr. Reg. No. 17,266 
Patrick McBride, Reg. No. 39,295 



Please direct al) communications relative to this application to the following addressee: 

WILUAWIJ. KUBIDA 
Holland & Hart llp 
Post Office Box 8749 
655 Seventeenth Street, Suite 3200 
Denver, CO 80201-8749 
Telephone (303) 296-8563 

ASSIGNEE 

Digital Equippjient Corporation 

Date; 10771^4^ 1^99 BY: 

NAME: Irene Kosturakis 

TITLE: Assistant Secretary / Assistant Clerk 

Irdftfl Kostumkitf. Aaalslant Se<ife(Ejry / Asaislam Ciork 

Autiiorizerf 7o Sign TWit Doei/mffnt On Bsha!/ Of Digiiai fiquipmanl Corporplmn 

Pursuiml To Board tt Difsclors R^solulion Datei Sepiombor 25. 1959 



